File System Event Tracking

ABSTRACT

Automated file system event tracking and reporting techniques are described in which file system events requested by a user application are intercepted and recorded prior to the request being permitted to pass to the file system for execution. Similarly, file system responses to a prior captured file system event are also intercepted and recorded. Predefined patterns of file system event may be aggregated and reported as a single event.

BACKGROUND

The invention relates generally to computer file systems and moreparticularly, but not by way of limitation, to methods and devices forrecording or tracking file system events.

In the context of computer operating systems, a “file” may be defined asa named collection of data. Files are normally retained in storagedevices such as magnetic disks (fixed, floppy, and removable), magnetictape, optical media such as compact disks (“CDs”) and digital videodisks (“DVDs”) and semiconductor memory devices such as electricallyprogrammable read-only memory (“EPROM”), electrically erasableprogrammable read-only memory (“EEPROM”), programmable gate arrays andflash devices.

A “file system” is that portion of an operating system whose primarytask is to manage the files retained on one or more storage devices. Thefile system is the means through which all files are manipulated (e.g.,created, destroyed and modified). To aid in this task, file systemsretain and/or obtain information about each file, so called “metadata.”Illustrative file metadata include the file's user-specified name, afile identifier (for uniquely identifying the file to the file system),a pointer or reference to the file in non-volatile storage (or mainmemory), the user ID associated with the file's creation, the time atwhich the file was created, the user ID associated with the lastmodification to the file, the time the last modification to the file wasmade and security information. Illustrative security informationincludes which specified user group or groups (e.g., administrator,employees and executives) are permitted to read or modify the file. Itwill be recognized that some, or all, of this metadata may be retainedwithin the file itself.

While prior art file system tracking applications provide some fileevent tracking capabilities, they do not compress or aggregate fileoperations for usability or track the user account making a change ordetect, identify. Thus, it would be beneficial to provide methods,applications and systems that provide these capabilities.

SUMMARY

Automated file system event tracking and reporting techniques aredescribed in which file system events requested by a user applicationare intercepted and recorded prior to the request being permitted topass to the file system for execution. Similarly, file system responsesto a prior captured file system event are also intercepted and recorded.Predefined patterns of file system event may be aggregated and reportedas a single event.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows, in flowchart format, a file tracking method in accordancewith one embodiment of the invention.

FIG. 2 shows a hierarchical directory structure.

FIG. 3 shows, in block diagram format, a file system tracking service inaccordance with one embodiment of the invention.

FIG. 4 shows typical file system events associated with file editoperations.

DETAILED DESCRIPTION

Methods and devices to provide real-time tracking of file system eventsand event aggregation reporting are described. The following descriptionis presented to enable any person skilled in the art to make and use theinvention as claimed and is provided in the context of embodimentsdesigned to operate in a Microsoft Windows® environment, variations ofwhich will be readily apparent to those skilled in the art. (MICROSOFTWINDOWS is a registered trademark of the Microsoft Corporation.)Accordingly, the claims appended hereto are not intended to be limitedby the disclosed embodiments, but are to be accorded their widest scopeconsistent with the principles and features disclosed herein.

Referring to FIG. 1, file tracking method 100 in accordance with oneembodiment of the invention waits for a file system event to occur(block 105). User-level file system requests are intercepted (block 110)and checked to determine if the request is of a specified type (block115). If the intercepted request is not of a type that has beenspecified to be tracked (the “No” prong of block 115), the request ispassed to the file system (block 125). If the request is of a typespecified to be tracked (the “Yes” prong of block 115), informationabout the request is recorded (block 120) and then allowed to pass tothe file system (block 125). When the file system completes processingthe event intercepted during the acts of block 110, its response isintercepted (block 130) and again checked to see if it is of a typespecified to be tracked (block 135). If the intercepted response is notassociated with a type of file system request that has been specified tobe tracked (the “No” prong of block 135), the response is passed to theappropriate application (block 145)—that is, the application making theinitial request. If the response is associated with file system requestof a type specified to be tracked (the “Yes” prong of block 135),information about the response is recorded (block 140) and then passedto the application that made the initial request (block 145).

Types of file system activity that may be specified by a user includedirectory and file operations such as create, delete, move, rename,security change, access denied while creating and access denied whileopening actions. In an embodiment designed to operate with the MicrosoftWindows operating system, these events correspond to the following typesof Microsoft defined file system events: IRP_MJ_CREATE;IRP_MJ_SET_INFORMATION; IRP_MJ_READ; IRP_MJ_WRITE; IRP_MJ_SET_SECURITY;IRP_MJ_CLEANUP; and IRP_MJ_CLOSE. In accordance with the invention, auser may specify that one or more of these file system actions betracked on a per-file, per-directory, per-user group, per-process orper-user identification basis. In addition, all of one or more specifiedevent types may also be tracked (regardless of file, directory, group,process or user identification). It will be recognized that file systemevent specification may be obtained from a user (e.g., a systemadministrator) though a graphical user interface (“GUI”) prior to usingtracking method 100. And, while the details of the graphical interfacemay vary from implementation to implementation, preparing such aninterface would be well within the skill of one of ordinary skill in theart.

Referring to the acts of blocks 120 (recording file system requestinformation) and 140 (recording file system response information), inone embodiment the following information is recorded: the time the eventoccurred; the user account identifier (e.g., the security identifier);they type of event (e.g., read, write, modify, create, delete, . . . );the local path of the affected file object (e.g., a file or directory);the process identifier; the name of the computer system on which theevent occurred; and security information. Security information includesboth “before” and “after” values. As used herein, a before valuecorresponds to the security state of an object at the time the filesystem event request was intercepted in accordance with block 110.Similarly, an after value corresponds to the security state of an objectat the time the file system event response was intercepted in accordancewith block 130. For example, if a file system event were to change adirectory from read/write access to read-only access, both of thesesecurity states would be recorded (one designated as the BEFORE statevalue and one designated as the AFTER state value). In anotherembodiment, the “difference” between the before and after securitystates may be recorded instead of separate before and after values. Anadvantage of this approach is that it provides a baseline to determinehow a file system object's security state is changed over time.Accordingly, methods in accordance with the invention permit securitychange histories to be obtained even if tracking operations areinterrupted (e.g., auditing is stopped for a period of time or an eventis missed).

In one embodiment both inherited and explicit security changes aretracked. When a security change to a specific file system object is made(e.g., a file or directory), this explicit change is noted and recorded.In addition, if a security change to one object is made (e.g., adirectory), that effects the security of hierarchically related objects,these changes are also determined and recorded. By way of example,consider FIG. 2 which shows hierarchical directory structure 200 havingroot directory 205, child directories 210 and 215, where directory 215itself has child directory 220. As shown, directory 210 includes fileobject 225, directory 215 includes file object 230 and directory 220includes file object 235. Consider a user-specified (explicit) securitychange to file object 225. Such a change is noted and recorded inaccordance with file tracking method 100. Now, consider an explicitsecurity change event directed to directory 215. This (explicit) changewould be noted and recorded in accordance with file tracking method 100and, in addition, if the explicit security change modifies an accessright to any hierarchically related object (e.g., directory object 220and/or file objects 230 or 235), this change too is noted and recorded(i.e., an inherited security change).

Referring to FIG. 3, tracking service 300 in accordance with oneembodiment of the invention includes intercept module 305, buffer 310,user-level module 315 and buffer 320. In general, operating system orkernel-mode module 305 intercepts file system requests (e.g., request325) form user-level applications (e.g., application 330) and filessystem responses (e.g., response 335) from file system 340. Aspreviously noted, service 300 may be initialized prior to beginningevent capture operations so that one or more file system events arespecified for tracking. As used herein, a “service” is a collection ofone or more code modules that do not require an active user session toexecute.

More specifically, intercept module 305 watches all file activity on agiven computer system such as, for example, a file server computersystem. If an intercepted file system request is identified as being ofa type designated for tracking, its information is temporarily stored inbuffer 310 after which it is passed to file system 340. Similarly,intercept module 305 watches all file system responses and, if aresponse is detected that corresponds to a prior stored request, itsinformation is also temporarily stored in buffer 310 after which it ispassed to the application making the initial request (e.g., application325). In one embodiment, user-level module 315 periodically polls buffer310 to retrieve recently captured file system activity information. Inanother embodiment, intercept module 305 periodically notifiesuser-level module 315 that there is event information cached in itsbuffer 310. In still another embodiment, intercept module 305 causes thecontents of buffer 310 to be periodically delivered to user-level module315. Regardless of which embodiment is actually implemented, after thecontents of buffer 310 is obtained by user-level module 315, interceptmodule 305 may reuse buffer 310.

For convenience, user-level module 315 stores file system eventinformation obtained from module 315/buffer 310 in buffer 315. Thisapproach permits a more rapid response to events coming “up” fromkernel-mode module 305 and, in addition, provides additional time foruser-level module 315 to process the collected information (seediscussion below). Once buffer 320 fills or after user-level module 315completes its processing, module 315 writes (identified by path 345) thecaptured and/or processed file system event information to an externalfile (e.g., file 350) on storage device 355. In one embodiment, file 350is a database file (e.g., a SQL database file). In another embodiment,file 350 is a flat file. In still another embodiment, file 350 is atemporary file that may later be incorporated into an event database orother event log.

Another aspect of file tracking method 100 is that user-level module 315may aggregate event information obtained from intercept module305/buffer 310. In prior art file system tracking procedures, eachindividual file system event captured is reported. In contrast, filetracking methods in accordance with the invention may intelligentlyaggregate captured events to reduce the number of reported events while,at the same time, reporting more user-relevant events. By way ofexample, consider typical office automation programs such as MicrosoftAccess,® Excel,® Powerpoint® and Word applications. (MICROSOFT ACCESS,EXCEL and POWERPOINT are registered trademarks of the MicrosoftCorporation.)

Referring to FIG. 4, in these and similar applications the file to beedited (e.g., original file 400) is copied into working memory fromstorage to create a working file such as, for example, work file 405(action designated by {circle around (1)}). A user, through theapplication, may then edit working file 405 (action designated by{circle around (2)}). When either the program or user determines it istime to “save” the file being edited, work file 405 is written fromworking memory to storage into work file 410 (action designated by{circle around (3)}), original file 400 is renamed to file 415 (actiondesignated by {circle around (4)}), work file 410 is renamed so that itsname is the same as that of the original file, 420 (action designated by{circle around (5)}) and, finally, original file 400 is deleted (actiondesignated by {circle around (6)}). In prior art file system trackingtechniques, the described “save” operation would have generated 4 eventrecords (actions {circle around (3)}, {circle around (4)}, {circlearound (5)} and {circle around (6)}). In contrast, file tracking method100 detects this pattern of activity (file A→B, followed by file C→A,followed by deleting file B) and, if it occurs within a designated timeperiod (e.g., between 0 and 3 seconds), records a single event: FileSave.

Similar aggregation processing may occur with other user-designated fileoperations such as, for example, file moves (copy file A to a newlocation, followed by deleting original file A) and file copyoperations. By way of example, a file open operation followed by one ormore read operations into a specified memory (e.g., buffer memory) and afile create operation in which the information used to “populate” thenew file comes from the same memory associated with the opened file canbe recognized as a file copy operation—regardless of the amount of timethe copy operation takes. Thus, in accordance with one embodiment of theinvention a potentially large number of file system events (file openevent+many, possibly thousands, of read events+file create event+many,possibly thousands, of write events where the read and write memory iscommon) may be conveniently collapsed or aggregated into a single eventlog: copy file A to file B by user C at time D. It will be recognizedthat such pattern recognition and event aggregation techniques arelacking form prior art file system tracking applications. It will befurther recognized in light of the current description that suchaggregation can eliminate up to 95% of the entries in a file systemtracking log by virtue of thousands of read and/or write operationsbeing associated with a single file operation (e.g., copy, move andrename).

One benefit of file tracking method 100 in accordance with the inventionis that it can reduce the number of events logged (through aggregatingevents identified though the detection of predefined patterns ofactivity) which, in turn, provides improved readability of activityreports. Another benefit in accordance with the invention is that itallows file tracking to be administered on any level desired (user,process, file or directory). Yet another benefit of the invention isthat user account information may be recorded. Still another benefit ofthe invention is that designated processes may be excluded for tracking.For example, it is known that virus monitoring software periodicallyreads substantially all files in a file system. If all such events wererecorded the system administrator would be inundated with irrelevantactivity (this is what prior art file system monitoring applicationsdo). In accordance with the invention, however, specified activities(e.g., file read events) of a specified user or process may be excluded.

Various changes in the materials and components as well as in thedetails of the illustrated operational methods are possible withoutdeparting from the scope of the following claims. For instance, filetracking methods in accordance with the invention may be used inoperational environments other than those running the Microsoft Windowsoperating system. In addition, not all of the information identified asbeing collected for a particular embodiment need be collected. In likemanner, more information then described herein may be captured. Inaddition, while file tracking method 100 has been described as being amonolithic entity (e.g., service 300), it will be recognized that inpractice it may comprise two or more separately executing threads. Forexample, a kernel thread may embody the functions described forintercept module 305 while a user thread may embody the functionsascribed to user-level module 315. Further, file tracking method 300does not have to be implemented as a “service.” That is, a file eventtracking method and system in accordance with the invention may run as astandard executable. That is, requiring an active user session toacquire and collect file system event information.

Acts in accordance with FIGS. 1 and 4 may be performed by a programmablecontrol device executing instructions organized into one or more programmodules. A programmable control device may be a single computerprocessor, a special purpose processor (e.g., a digital signalprocessor, “DSP”, a plurality of processors coupled by a communicationslink or a custom designed state machine. Custom designed state machinesmay be embodied in a hardware device such as an integrated circuitincluding, but not limited to, application specific integrated circuits(“ASICs”) or field programmable gate array (“FPGAs”). Storage devicessuitable for tangibly embodying program instructions include, but arenot limited to: magnetic disks (fixed, floppy, and removable) and tape;optical media such as CD-ROMs and digital video disks (“DVDs”); andsemiconductor memory devices such as Electrically Programmable Read-OnlyMemory (“EPROM”), Electrically Erasable Programmable Read-Only Memory(“EEPROM”), Programmable Gate Arrays and flash devices.

1. A file system tracking method, comprising: intercepting a file systemsecurity change request directed to a target file system object, thetarget file system object having a current security state, the securitychange request specifying a final security state; recording the currentsecurity state; communicating the security change request to a filesystem; intercepting an indication that the security change request hasbeen processed by the file system; and recording the final securitystate.
 2. The method of claim 1, wherein the acts of intercepting areperformed by a kernel-level application.
 3. The method of claim 1,wherein the security change request comprises a IRP_MJ_SET_SECURITYrequest.
 4. The method of claim 1, wherein the act of recording thefinal security state comprises recording a single datum from which boththe current security state and final security state may be derived. 5.The method of claim 1, wherein the act of recording the current securitystate comprises storing the current security state and an identifieridentifying the target file system object in a kernel memory.
 6. Themethod of claim 5, wherein the act of recording the final security statecomprises associating the final security state with the current securitystate in the kernel memory.
 7. The method of claim 6, furthercomprising: retrieving the current security state, the target securitystate and the identifier identifying the target file system object fromthe kernel buffer memory; and storing the current security state, thetarget security state and the identifier identifying the target filesystem object in a database.
 8. The method of claim 7, wherein the actof retrieving and storing is performed by a user-level application. 9.The method of claim 1, wherein the act of recording the current securitystate and the final security state further comprises recording a timeassociated with the intercepted file system security change request. 10.The method of claim 1, wherein the act of recording the current securitystate and the final security state further comprises recording a useraccount identifier associated with the security change request.
 11. Themethod of claim 1, wherein the act of recording the current securitystate and the final security state further comprises recording a processidentifier associated with the security change request.
 12. The methodof claim 1, wherein the act of recording the current security state andthe final security state further comprises recording a computer systemidentifier on which the target file system object is stored.
 13. Themethod of claim 1, wherein the act of recording the current securitystate and the final security state further comprises recording a pathassociated with the target file system object.
 14. A file systemtracking method, comprising: identifying a first file system eventchanging a first file system object's name from a first name to a secondname; identifying a second file system event changing a second filesystem object's name to the first name; and aggregating the first andsecond file system events as a single file modification event for thefirst file system object.
 15. The method of claim 14 further comprising,prior to the act of aggregating, identifying a file system delete eventfor the first file system object.
 16. The method of claim 14, whereinthe act of identifying the first file system event comprises determininga first time associated with the first file system event, the actidentifying the second file system event comprises determining a secondtime associated with the second file system event, and the act ofaggregating occurs only if the time between the first and second timesis within a specified range.
 17. The method of claim 16, wherein thespecified range is approximately 2 seconds.
 18. The method of claim 14,wherein the acts of identifying and aggregating are performed by auser-level application.
 19. The method of claim 14, wherein the act ofaggregating further comprises writing the file modification event to alog file.
 20. The method of claim 19, wherein the act of writing thefile modification event to a log file comprises writing the filemodification to a database.
 21. The method of claim 20, wherein the actof writing the file modification event to a database comprises writingthe file modification event to a relational database.
 22. The method ofclaim 21, wherein the database comprises a SQL database.
 23. A filesystem audit method, comprising: intercepting, by a kernel levelapplication, file system events; recording the intercepted file systemevents in a kernel memory; retrieving the recorded file system eventsfrom kernel memory to a user level memory; identifying, from theretrieved file system events, a first file system event changing a firstfile system object's name from a first name to a second name;identifying, from the retrieved file system events, a second file systemevent changing a second file system object's name to the first name;consolidating the first and second file system events into a third filesystem event; and recording the third file system event as a filemodification event for the first file system object.
 24. The method ofclaim 23 further comprising, prior to the act of consolidating,identifying, from the retrieved file system events, a file system deleteevent for the first file system object.
 25. The method of claim 23,wherein the act of identifying the first file system event comprisesdetermining a first time associated with the first file system event,the act identifying the second file system event comprises determining asecond time associated with the second file system event, and the act ofconsolidating occurs only if the time between the first and second timesis within a specified range.
 26. The method of claim 25, wherein thespecified range is approximately 2 seconds.
 27. The method of claim 23,wherein the act of recording comprises writing to a log.
 28. The methodof claim 23, wherein the act of writing to a log comprises writing to adatabase.
 29. The method of claim 28, wherein the database comprises aSQL database.
 30. A file system audit method, comprising: intercepting,by a kernel level application, file system events; recording theintercepted file system events in a kernel memory; retrieving therecorded file system events from kernel memory to a user level memory;identifying, from the retrieved file system events, a first file systemevent copying a first file system object from a first location to asecond location; identifying, from the retrieved file system events, asecond file system event deleting the first file system object from thefirst location; consolidating the first and second file system eventsinto a third file system event; and recording the third file systemevent as a file move event for the first file system object.
 31. Themethod of claim 30, wherein the act of consolidating occurs only if atime associated with the first file system event and a time associatedwith the second file system event within a specified time interval. 32.The method of claim 31, wherein the specified time interval isapproximately 2 seconds.
 33. A file system tracking method, comprising:identifying a first file system event opening a first file systemobject; identifying a second file system event creating a second filesystem object; detecting one or more read operations directed to thefirst file system object into a specified memory; detecting one or morefile system write operations to the second file system object from thespecified memory; and consolidating the first file system event, thesecond file system event, the one or more file system read operationsand the one or more file system write operations into a single filemodification event.
 34. The method of claim 33, wherein the act ofidentifying the first file system event comprises determining a firsttime associated with the first file system event, the act identifyingthe second file system event comprises determining a second timeassociated with the second file system event, and the act ofconsolidating occurs only if the time between the first and second timesis within a specified range.
 35. The method of claim 34, wherein thespecified range is approximately 2 seconds.
 36. The method of claim 33,wherein the acts of identifying and consolidating are performed by auser-level application.
 37. The method of claim 33, wherein the act ofconsolidating further comprises writing the file modification event to alog file.
 38. The method of claim 37, wherein the act of writing thefile modification event to a log file comprises writing the filemodification to a database.
 39. The method of claim 38, wherein the actof writing the file modification event to a database comprises writingthe file modification event to a relational database.
 40. A programstorage device, readable by a programmable control device, comprisinginstructions stored on the program storage device for causing theprogrammable control device to perform the method of claim
 14. 41. Aprogram storage device, readable by a programmable control device,comprising instructions stored on the program storage device for causingthe programmable control device to perform the method of claim
 1. 42. Aprogram storage device, readable by a programmable control device,comprising instructions stored on the program storage device for causingthe programmable control device to perform the method of claim
 23. 43. Aprogram storage device, readable by a programmable control device,comprising instructions stored on the program storage device for causingthe programmable control device to perform the method of claim
 30. 44. Aprogram storage device, readable by a programmable control device,comprising instructions stored on the program storage device for causingthe programmable control device to perform the method of claim 33.